Provable convergence of Nesterov’s accelerated gradient method for over-parameterized neural networks

Momentum methods, such as heavy ball method (HB) and Nesterov’s accelerated gradient (NAG), have been widely used in training neural networks by incorporating the history of gradients into current updating process. In practice, they often provide improved performance over (stochastic) descent (GD) with faster convergence. Despite their empirical success, a theoretical understanding convergence rates is still insufficient. Recently, some attempts made analyzing trajectories gradient-based methods an over-parameterized regime, where number parameters significantly larger than that instances. However, majority existing work mainly concerned GD established result NAG inferior to HB GD, which fails explain practical success NAG. this paper, we take step towards closing gap randomly initialized two-layer fully connected network ReLU activation. fact objective function non-convex non-smooth, show converges global minimum at non-asymptotic linear rate (1−Θ(1/κ))t, κ>1 condition gram matrix t iterations. Compared (1−Θ(1/κ))t our provides guarantees for acceleration training. Furthermore, findings suggest similar rate. Finally, extensive experiments on six benchmark datasets conducted validate correctness results.